Automatic learning of dialogue strategy using dialogue simulation and reinforcement learning

This paper describes a method for automatic design of human-computer dialogue strategies by means of reinforcement learning, using a dialogue simulation tool to model the user behaviour and system recognition performance. To the authors’ knowledge this is the first application of a detailed simulation tool to this problem. The simulation tool is trained on a corpus of real user data. Compared to direct state transition modelling, it has the major advantage that different state space representations can be studied without collecting more training data. We applied Q-learning with eligibility traces to obtain policies for a telephone-based cinema information system, comparing the effect of different state space representations and evaluation functions. The policies outperformed handcrafted policies that operated in the same restricted state space, and gave performance similar to the original design that had been through several iterations of manual refinement.